On measures of agreement calculated from contingency tables with categories defined by the empirical quantiles of the marginal distributions.

Epidemiologists sometimes collect bivariate continuous data on a number of subjects, compute the empirical (sample) quantiles of the marginal data, and then use these values to partition the original data into two-way contingency tables. Tables created in this manner have row and column categories defined by the random empirical marginal quantiles rather than by preset cutpoints, so these tables have fixed marginal totals. Hence, instead of the conventional multinomial distribution, these tables have the empirical bivariate quantile-partitioned (EBQP) distribution. In this paper, the authors demonstrate how to use empirical methods appropriate for EBQP tables to make inferences and construct confidence intervals for three commonly used measures of agreement: kappa, weighted kappa, and another class of measures derived from conditional proportions in the extreme rows of the table. They also show that if one incorrectly applies conventional methods appropriate for multinomial tables to statistics calculated from EBQP tables, one can obtain substantially misleading results. In addition, the authors present alternative parametric methods for estimating these measures of agreement and illustrate corresponding methods of inference and confidence interval construction. Finally, they show that these empirical (EBQP) methods can have low efficiency compared with parametric methods for some of these measures of agreement.